1. Field of the Invention
This invention relates to an image processing method and apparatus. This invention particularly relates to an improvement in an image processing method and apparatus, wherein only a specific image portion, such as an abnormal pattern, which is embedded in an image, is extracted from the image and emphasized selectively.
2. Description of the Prior Art
Image processing, such as gradation processing or frequency processing, has heretofore been carried out on an image signal (i.e., an original image signal), which represents an original image having been obtained with one of various image obtaining methods, such that a visible image having good image quality can be reproduced and used as an effective tool in, particularly, the accurate and efficient diagnosis of an illness. Particularly, in the field of medical images, such as radiation images of human bodies serving as objects, it is necessary for specialists, such as doctors, to make an accurate diagnosis of an illness or an injury of the patient in accordance with the obtained image. Therefore, it is essential to carry out the image processing in order that a visible image having good image quality can be reproduced and used as an effective tool in the accurate and efficient diagnosis of an illness.
As one of the image processing, frequency emphasis processing has been disclosed in, for example, Japanese Unexamined Patent Publication No. 61(1986)-169971. With the disclosed frequency emphasis processing, an original image signal Sorg, which may represent the image density value, or the like, of an original image is converted into a processed image signal Sproc with Formula (1) shown below. EQU Sproc=Sorg+.beta..times.(Sorg-Sus) (1)
In Formula (1), .beta. represents the frequency emphasis coefficient, and Sus represents the unsharp mask signal. The unsharp mask signal Sus comprises super-low frequency components obtained by setting a mask, i.e. an unsharp mask, constituted of a picture element matrix, which has a size of N columns.times.N rows (wherein N represents an odd number) and has its center at the picture element represented by the original image signal Sorg, in a two-dimensional array of picture elements of the image. The unsharp mask signal Sus is calculated with, for example, Formula (2) shown below. EQU Sus=(.SIGMA..SIGMA.Sorg)/N.sup.2 ( 2)
wherein .SIGMA..SIGMA.Sorg represents the sum of the image signal values representing the picture elements located within the unsharp mask.
The value of (Sorg-Sus) in the parenthesis of the second term of Formula (1) is obtained by subtracting the unsharp mask signal Sus, which represents the super-low frequency components, from the original image signal Sorg. Therefore, the value of (Sorg-Sus) represents the comparatively high frequency components, which have been extracted selectively by eliminating the super-low frequency components from the original image signal Sorg.
The comparatively high frequency components are then multiplied by the frequency emphasis coefficient .beta., and the obtained product is added to the original image signal Sorg. In this manner, of the original image, only the comparatively high frequency components can be selectively and relatively subjected to emphasis or restriction (adjustment of sharpness).
Also, processing based upon the algorithm of morphology (hereinbelow referred to as the morphology operation or the morphology processing) has heretofore been known as the operation processing for selectively extracting only a specific image portion, such as an abnormal pattern, or an image edge portion from an original image.
The morphology processing has been studied as a technique efficient for detecting, particularly, a small calcified pattern, which is one of characteristic forms of mammary cancers. However, the image to be processed with the morphology processing is not limited to the small calcified pattern in a mammogram, and the morphology processing is applicable to any kind of image, in which the size and the shape of a specific image portion (i.e., an abnormal pattern, or the like) to be detected are known previously.
The morphology processing is carried out by using a structure element (also referred to as a mask) B, which is set in accordance with the size of the image portion to be extracted, and a multi-scale .lambda.. The morphology processing has the features in that, for example, (1) it is efficient for extracting a calcified pattern itself, (2) it is not affected by complicated background information, and (3) the extracted calcified pattern does not become distorted.
Specifically, the morphology processing is advantageous over ordinary differentiation processing in that it can more accurately detect the geometrical information concerning the size, the shape, and the image density distribution of the calcified pattern. How the morphology processing is carried out will be described hereinbelow by taking the detection of a small calcified pattern in a mammogram as an example.
Fundamental operation of morphology processing
In general, the morphology processing is expanded as the theory of sets in an N-dimensional space. As an aid in facilitating the intuitive understanding, the morphology processing will be described hereinbelow with reference to a two-dimensional gray level image.
The gray level image is considered as a space, in which a point having coordinates (x, y) has a height corresponding to an image density value f(x, y). In this case, it is assumed that the image signal representing the image density value f(x, y) is a high luminance-high signal level type of image signal, in which a low image density (i.e., a high luminance when the image is displayed on a CRT display device) is represented by a high image signal level.
Firstly, as an aid in facilitating the explanation, a one-dimensional function f(x) corresponding to the cross-section of the two-dimensional gray level image is considered. It is assumed that a structure element g (corresponding to the aforesaid structure element B) used in the morphology operation is a symmetric function of Formula (3) shown below, which is symmetric with respect to the origin. EQU g.sup.s (x)=g(-x) (3)
It is also assumed that the value is 0 in a domain of definition G, which is represented by Formula (4). EQU G={-m, -m+1, . . . , -1, 0, 1, . . . , m-1, m} (4)
In such cases, the fundamental forms of the morphology operation are very simple operations carried out with Formulas (5), (6), (7), and (8) shown below. EQU dilation; f.sym.G.sup.s !(i)=max{f(i-m), . . . ,f(i), . . . ,f(i+m)}(5) EQU erosion; f.crclbar.G.sup.s !(i)=min{f(i-m), . . . ,f(i), . . . ,f(i+m)}(6) EQU opening; f.sub.g =(f.crclbar.g.sup.s).sym.g (7) EQU closing; f.sup.g =(f.sym.g.sup.s).crclbar.g (8)
Specifically, as illustrated in FIG. 8A, the dilation processing is the processing for retrieving the maximum value in the region of a width of .+-.m (which width is the value determined in accordance with the structure element B and corresponds to the mask size shown in FIG. 8A) having its center at a picture element of interest. As illustrated in FIG. 8B, the erosion processing is the processing for retrieving the minimum value in the region of the width of .+-.m having its center at the picture element of interest. Therefore, the dilation processing is also referred to as the maximum value processing, and the erosion processing is also referred to as the minimum value processing.
The opening processing is equivalent to the processing in which the dilation processing is carried out after the erosion processing, i.e., the processing in which the maximum value is searched after the searching of the minimum value. Also, the closing processing is equivalent to the processing in which the erosion processing is carried out after the dilation processing, i.e., the processing in which the minimum value is searched after the searching of the maximum value.
More specifically, as illustrated in FIG. 8C, the opening processing is equivalent to the processing for smoothing the image density curve f(x) from the low luminance side, and removing a convex image density fluctuating portion (i.e., the portion at which the luminance is higher than that of the surrounding portions), which fluctuates in a region spatially narrower than the mask size of 2 m. Also, as illustrated in FIG. 8D, the closing processing is equivalent to the processing for smoothing the image density curve f(x) from the high luminance side, and removing a concave image density fluctuating portion (i.e., the portion at which the luminance is lower than that of the surrounding portions) which fluctuates in the region spatially narrower than the mask size of 2 m.
In cases where the structure element g is not symmetric with respect to the origin, the dilation operation with Formula (5) is referred to as the Minkowski sum, and the erosion operation with Formula (6) is referred to as the Minkowski difference.
In cases where the image signal representing the image density value f(x) is a high image density-high signal level type of image signal, in which a high image density is represented by a high image signal level, the relationship between the image density value f(x) and the image signal value becomes reverse to the relationship between the image density value f(x) and the image signal value in the high luminance-high image signal level type of image signal. Therefore, the dilation processing, which is carried out on the high image density-high signal level type of image signal, coincides with the erosion processing, which is carried out on the high luminance-high signal level type of image signal as shown in FIG. 8B. The erosion processing, which is carried out on the high image density-high signal level type of image signal, coincides with the dilation processing, which is carried out on the high luminance-high signal level type of image signal as shown in FIG. 8A. The opening processing, which is carried out on the high image density-high signal level type of image signal, coincides with the closing processing, which is carried out on the high luminance-high signal level type of image signal as shown in FIG. 8D. Also, the closing processing, which is carried out on the high image density-high signal level type of image signal, coincides with the opening processing, which is carried out on the high luminance-high signal level type of image signal as shown in FIG. 8C.
The morphology processing is herein described with respect to the high luminance-high signal level type of image signal.
Application to detection of calcified patterns
In order for a calcified pattern to be detected, it is considered to employ a difference method, in which a smoothed image signal is subtracted from the original image signal. However, with a simple smoothing method, it is difficult to discriminate the calcified pattern from an elongated non-calcified pattern (for example, a pattern of the mammary gland, a blood vessel, mammary gland supporting tissues, or the like). Therefore, Obata of Tokyo University of Agriculture and Technology, et al. have proposed a morphology filter, which is represented by Formula (9) and is based upon the opening operation using a multiply structure element. Reference should be made to "Extraction of Small Calcified Patterns with A Morphology Filter Using A Multiply Structure Element," Collected Papers of The Institute of Electronics and Communication Engineers of Japan, D-II, Vol. J75-D-II, No. 7, pp. 1170-1176, July 1992; and "Fundamentals of Morphology and Its Application to Mammogram Processing," Medical Imaging Technology, Vol. 12, No. 1, January
______________________________________ P = f-max {(f .crclbar. Bi) .sym. Bi} i.epsilon.(1, . . . , M) = f-max {f.sub.Bi } i.epsilon.(1, . . . , M) (9) ______________________________________
In Formula (9), Bi (wherein i=1, 2, . . . , M) represents M number of linear structure elements (M=4 in the example shown in FIG. 9). (The M number of structure elements, as a whole, will hereinbelow be referred to as the multiply structure element.) In cases where the structure element Bi is set to be larger than the calcified pattern to be detected, a calcified pattern, which is a convex signal change portion finer than the structure element Bi (i.e., which is an image portion fluctuating in a spatially narrow region) and has luminance values larger than the luminance values of the surrounding portions, is removed in the opening processing. On the other hand, an elongated non-calcified pattern is longer than the structure element Bi. Therefore, in cases where the inclination of the non-calcified pattern (i.e, the direction along which the non-calcified pattern extends) coincides with one of the directions of the four structure elements Bi, the non-calcified pattern remains unremoved after the opening processing, i.e. the operation of the second term of Formula (9), has been carried out. Therefore, when the smoothed image signal obtained from the opening processing (i.e. the signal representing the image, from which only the calcified pattern has been removed) is subtracted from the original image signal f, an image can be obtained which contains only the small calcified pattern. This is the concept behind Formula (9).
As described above, in cases where the image signal is of the high image density-high signal level type, the image density value of the calcified pattern is smaller than the image density values of the surrounding image portions, and the calcified pattern constitutes a concave signal change portion with respect to the surrounding portions. Therefore, the closing processing is applied in lieu of the opening processing, and Formula (10) shown below is applied in lieu of Formula (9).
______________________________________ P = f-min {(f .sym. Bi) .crclbar. Bi} i.epsilon.(1, . . . , M) = f-min {f.sub.Bi } i.epsilon.(1, . . . , M) (10) ______________________________________
Also, in cases where the image signal obtained from the dilation processing or the erosion processing is subtracted from the original image signal in accordance with Formula (11) or (12) shown below, only the edge portion in the original image (i.e., the hatched portion in FIGS. 8A or 8B) can be extracted selectively. EQU P=f-(f.sym.Bi) (11) EQU P=f-(f.crclbar.Bi) (12)
As described above, in order that a visible image having good image quality can be reproduced and used as an effective tool in, particularly, the accurate and efficient diagnosis of an illness, it is essential to carry out the image processing on the given image. However, in cases where the emphasis processing merely depending on the image density is carried out as disclosed in, for example, U.S. Pat. No. 4,315,318, components adversely affecting the image quality, such as radiation noise components in a mammogram, are also emphasized. As a result, the image quality of the image and its capability of serving as an effective tool in, particularly, the efficient and accurate diagnosis of an illness become low.
Also, as disclosed in, for example, U.S. Pat. No. 4,571,635, EP 357842 B1, and EP 406382 B1, in cases where emphasis processing depending upon the value of variance of an image signal is carried out, an image portion having a locally large change in image density is emphasized to a high extent. Therefore, the problems occur in that undershooting and overshooting become relatively perceptible in the vicinity of the image portion. Particularly, as for X-ray images, an artifact is apt to occur on the high image density side.
Accordingly, the applicant proposed image processing methods, wherein emphasis depending upon the morphology signal obtained from each morphology operation described above is carried out, wherein components unnecessary for a diagnosis, or the like, such as noise components, are not emphasized, and wherein only a specific image portion of interest is emphasized efficiently. The proposed image processing methods are described in U.S. Ser. No. 08/623,223.
With the proposed image processing methods, only the calcified pattern having a size smaller than the structure element can be selectively extracted as the image portion of interest. Therefore, in cases where the emphasis processing depending upon the signal representing the extracted image portion is carried out, it is possible to obtain good effects of efficiently emphasizing only the calcified pattern, which effects could not be obtained in the past.
However, with the proposed image processing methods, it rarely occurs that an unnatural emphasized image is obtained.
FIGS. 10A-1, 10A-2, and 10A-3 are explanatory views showing how the results of processing are obtained from the processing carried out on an original image signal representing a calcified pattern smaller than a structure element. FIGS. 10B-1, 10B-2, and 10B-3 are explanatory views showing how the results of processing are obtained from the processing carried out on an original image signal representing an image pattern, that is larger than a structure element and contains a flat portion, at which the fluctuation in the signal value is comparatively small and which has a size smaller than the structure element. FIGS. 10C-1, 10C-2, and 10C-3 are explanatory views showing how the results of processing are obtained from the processing carried out on an original image signal representing an image pattern, that is larger than a structure element and contains a flat portion, at which the fluctuation in the signal value is comparatively small and which has a size sufficiently larger than the structure element.
Specifically, as illustrated in FIG. 10A-2, in cases where the closing processing is carried out as the morphology operation on the original image signal shown in FIG. 10A-1, which represents the calcified pattern smaller than the structure element, the profile of the original image signal is smoothed completely. Also, as illustrated in FIG. 10C-2, in cases where the closing processing is carried out on the original image signal of FIG. 10C-1 representing the image pattern, that is larger than the structure element and contains the flat portion, at which the fluctuation in the signal value is comparatively small and which has a size sufficiently larger than the structure element, the profile of the original image signal is kept unchanged. Therefore, as for the calcified pattern smaller than the structure element, as illustrated in FIG. 10A-3, when the signal obtained from the closing operation is subtracted from the original image signal, a signal having signal values other than 0 is obtained. On the other hand, as for the image pattern, that is larger than the structure element and contains the flat portion, at which the fluctuation in the signal value is comparatively small and which has a size sufficiently larger than the structure element, when the signal obtained from the closing operation is subtracted from the original image signal, a signal having a signal value of 0 is obtained. Accordingly, the calcified pattern of FIG. 10A-1 and the image pattern of FIG. 10C-1 can be discriminated from each other.
However, as illustrated in FIG. 10B-2, in cases where the closing processing is carried out on the original image signal of FIG. 10B-1 representing the image pattern, that is larger than the structure element and contains the flat portion, at which the fluctuation in the signal value is comparatively small and which has a size smaller than the structure element, the portion at which the signal value fluctuates in the region smaller than the structure element (i.e., the portion corresponding to the center area of the image pattern) is smoothed as in the calcified pattern of FIG. 10A-1 smaller than the structure element. Therefore, as illustrated in FIG. 10B-3, when the signal obtained from the closing operation is subtracted from the original image signal, the signal value becomes equal to 0 at the part corresponding to the slope area, which surrounds the center area in the image pattern, and signal values other than 0 are obtained at the part corresponding to the center area, which is located on the side inward from the slope area in the image pattern. As a result, as for the image pattern shown in FIG. 10B-1, only the region, which contains the center area of the image pattern and is smaller than the structure element, is emphasized.